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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply Is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even If timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )D Responsive to communication(s) filed on 07 December 2003 . 
2a)^ This action is FINAL. 2b)n This action is non-final. 

3) n Since this application is in condition for allowance except for fornnai matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 0,G. 213. 
Disposition of Claims 

4) D Claim(s) 1,2,4 and 6-15 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) 0 Claim(s) 1,2,4,6-10 and 12-15 is/are reiected. 

Claim(s) 1± is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner. 

10)n The drawing(s) filed on is/are: a)n accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
1 1 )□ The proposed drawing correction filed on is: a)\Z\ approved b)\Z\ disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) n The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

13) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)nAII b)n Some*c)n None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) D The translation of the foreign language provisional application has been received. 

1 5) n Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 1 20 and/or 121. 
Attachment(s) 

1 ) □ Notice of References Cited (PTO-892) 4) O Interview Summary (PTO-41 3) Paper No(s). 



2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) C] Notice of Infomial Patent Application (PTO-152) 

3) □ InfonmaUon Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 

1 . This Office Action is in response to Request for Reconsideration entered 
12/7/2003 for the application (09/488.028), filed on 1/20/2000. 

2. The pending claims 1, 2, 4, 6-15 are examined. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

3. Claims 1, 2, 4, 6, 7, 12-15 are rejected under 35 U.S.C, 102(a) as being 
anticipated by Brondsted, et a! The IntelliMedia WorkBench A Generic Environment 
For Multimodal Systems," (1998). 

With regard to claim 1 : 

As per "a method of locating and displaying an image of a target," Brondsted 
describes a method of locating and displaying an image of a target (see fig. 1 ); 

As per "sensing a triggering event generated by a human operator;" Brondsted 
describes sensing spoken word (key word or command) as well as user's gesture via a 
microphone and camera respectively (see section 3); 

As per "receiving additional external information that characterizes at least one 
machine-sensible feature of a target, said receiving step occurring substantially 
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simultaneously with said sensing step;" since Brondsted is a multimodal system, thus 
additional information about a target or location can be received through spoken word 
(extracted key word) input as well as through gesture input (section 3). These inputs are 
executed simultaneously (section 2.1); 

As per "aiming a camera in response to said sensing and said receiving step, 
wherein said sensing step includes sensing a gesture indicting a direction of said 
target," Brondsted describes a simultaneous speech and gesture input implemented on 
Workbench (see section 2.1). Brondsted further describes and illustrates (fig. 1) a 
camera directed toward the target; wherein the camera continuously captures the 
pointing hand over the workbench while the user/operator describes the location 
(section 2.1). 
With regard to claim 2 : 

As per "... said sensing step includes sensing a gesture of a human operator 
indicating a target." Brondsted discloses Gesture recognizer (fig. 2) for sensing a 
gesture of a human operator indicating a target (see Brondsted, fig. 1). 
With regard to claim 4 : 

As per "... said receiving step includes receiving speech from said human 
operator." Brondsted discloses Microphone (fig. 2) for receiving speech from said 
human operator (see Brondsted, section 2.1). 
With regard to claim 6 : 

As per "... processing said speech for use with at least one machine sensor, said 
at least one machine sensor and said speech assisting in locating said target." 
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Brondsted disclose Speecli recognizer, Speech synthesizer, and IVIicroscope (see 
Brondsted, fig. 2, and section 2.1). 
With regard to claim 7 : 

As per "... said sensing step includes sensing a gesture indicting a direction from 
said human operator to said target," Brondsted discloses a gesture indicating a direction 
form said human operator to said target (see Brondsted, fig. 1). 
With regard to claim 12: 

As per "A method of locating and displaying an image of a target," Brondsted 
describes a method of locating and displaying an image of a target (see fig. 1); 

As per "scanning an area within the range of at least one sensor;" Brondsted 
illustrates scanning a workbench (area) with a camera and laser pointer (fig. 1); 
As per "identifying potential targets;" Brondsted describes identifying location (target) on 
the workbench shown with a campus information application (see section 2); 

As per "storing information concerning machine sensible characteristics and 
locations of said possible targets;" Brondsted describes storing the multimodal campus 
information (section 2.1 ). 

As per "sensing a triggering event, said triggering event generated by a human 
operator;" Brondsted describes sensing spoken word (key word or command as well as 
user's gesture via a microphone and camera respectively (see section 3); 

As per "receiving additional external information that characterizes at least one 
feature of said target, said receiving step occurring substantially simultaneously with 
said sensing step;" since Brondsted is a multimodal system, thus additional information 
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about a target or location can be received through spoken word (extracted keyword) 
input as well as through gesture input (section 3). These inputs are executed 
simultaneously (section 2.1); and 

As per "aiming a camera in response to said sensing, storing and said receiving 
steps, wherein said sensing step includes sensing a gesture Indicting a direction of said 
target." Brondsted describes a simultaneous speech and gesture input implemented on 
Workbench (see section 2.1). Brondsted further describes and illustrates (fig. 1) a 
camera directed toward the target; wherein the camera continuously captures the 
pointing hand over the workbench while the user/operator describes the location 
(section 2.1). 
With regard to claim 13: 

As per "A method of aiming a camera at a target," Brondsted illustrates aiming a 
camera and a laser pointer at a campus map location (target) (fig. 1 ). 

As per "inputting an indication of a position of a target;" Brondsted illustrates and 
describes pointing toward a location of a target (fig. 1 , see also section 3); 

As per " inputting further information about a machine-sensible characteristic of 
said target;" Brondsted describes sensing spoken word (key word or command) as well 
as user's gesture via a microphone and camera respectively (see section 3); 

As per "aiming a camera at said target in response to said indication and said 
further information to reduce an error in said aiming, wherein said inputting an indication 
step includes inputting a gesture indicting a direction of said target." Brondsted 
describes a camera and a laser pointer directed or focused toward a displayed 
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workbench application. Brondsted further describes an automatic calibration procedure 
involving both the camera and laser pointer. Using the camera the gesture recognizer is 
able to track 2D pointing gestures in real time. The camera continuously captures 
images. The hand gesture or the hand motion is analyzed in order to find the direction 
of the pointing hand (device) and its edge. 
With regard to claim 14: 

Claim 14, while not necessary identical in scope, contain limitations similar to 
independent claim 13 and therefore are rejected under the same rationale. 
With regard to claim 15: 

As per "...said step of orienting includes orienting a camera." Brondsted, as 
illustrated in fig. 1 shows oriented camera view toward a workbench. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 8-10 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Tom Brondsted, et al "The IntelliMedia WorkBench A Generic Environment For 
Multimodal Systems," (1998) in view of Indrajit Poddar, et al "Toward Natural 
Gesture/Speech HCI: A Case Study of Weather Narration," (1998). 
With regard to claim 8: 
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As per "...said processing step includes processing said voice information 
through a look-up table corresponding said speech to search criteria for use with said at 
least one sensor." Brondsted does describe different module and for storing data, but 
Brondsted fails to describe "processing said voice information through a look-up table 
corresponding to said speech to search criteria for use with said at least one sensor." 
Poddar discloses a multimodal system, including speech (via Microphone) and gesture 
(hand) input (section 3). Poddar further discloses processing voice information through 
a look-up table (tablel- table 4). Therefore it would have been obvious as the time the 
invention was made to replace Brondsted's voice information memory with Poddar's 
look-up table because it would be easier to structure the voice information and access 
the voice information as a table format. 
With regard to claim 9 : 

As per "... said look-up table is modifiable." Brondsted in view of Poddar further 
describe replacing key words of the table, modifiable look-up table (Poddar, section 3). 
With regard to claim 10 : 

As per "...said look-up table modifiable by receiving information through the on- 
line global compute network. " since Brondsted system can be implemented in a 
distributed environment (sections 2.1-2.2), the look-up table (voice data memory 
module) could be modified by information received from other remote devices. 



Allowable Subject Matter 
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5. Claim 1 1 is objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the 
base claim and any intervening claims. 

The following is a statement of reasons for the indication of allowable subject 
matter: Although Brondsted and Poddar describes a modifiable look-up table (poddar, 
section 3) that includes replaces word or phrase input with another input and a 
corresponding search criteria (Poddar, section 3), " said added voice input and said 
corresponding search criteria established by comparing previous association of said 
added voice input with at least one machine sensible characteristic of at least one 
correctly identified target associated with said voice input, said machine sensible 
characteristic being a basis for determining said corresponding search criteria." not 
clearly described. 

Response to Arguments 

6. Applicant's arguments filed 12/7/2003 have been fully considered but they are 
not persuasive. Applicant argues, "the spoken query inputs in this [2.1] section of 
Brondsted do not teach "receiving additional external information that characterizes at 
least one machine-sensible feature of a target." However, in section 2.1 , Brondsted 
describes the multimodal system receives input from both speech and/or gesture 
simultaneously. Thus, the examiner believes that in addition to a pointing gesture 
recognition, the user can issue spoken commands or key words (additional external 
information) to the speech recognition system, then, the multimodal system of 
Brondsted outputs a synchronized pointing gesture and/or speech (issued spoken 
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commands) (see sections 2.1-2.2). Thus, Brondsted describes "receiving additional 
external information that characterizes at least one machine-sensible feature of a 
target." Applicant further argues Brondsted fails to teach at least orienting an 
instrument with respect to said target to acquire said target in response to said spatial 
information and further information to reduce an ambiguity in said position." In contrast, 
Brondsted describes simultaneous inputs of speech and /or gesture to reduce the 
ambiguity when acquiring a target (see sections 2.1-2.2). 

Furthermore, the applicant argues that Poddar does not include teachings that cure the 
deficiencies of Brondsted. In contrast to the applicant's argument, Poddar, as in 
Brondsted, teaches a multimodal system including the claimed look-up table (see table 



7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 



1-4). 



Conclusion 
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8. Any inquiry concerning this communication or earlier communications from the 
Examiner should be directed to Tadesse Hailu, whose telephone number is (703) 306- 
2799. The Examiner can normally be reached on M-F from 10:00 - 6:30 ET. If attempts 
to reach the Examiner by telephone are unsuccessful, the Examiner's supervisor, John 
Cabeca, can be reached at (703) 308-31 1 6 Art Unit 21 73 CPK 2-4A51 . 

9. Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the Group receptionist whose telephone number Is 
(703) 305-3900. 



1/13/2004 




JOHN CABECA 
SUPERVISORY PATENT EXAMINFP 
TECHNOLOGY CENTER 21 CO 



